{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "db0afeeb",
   "metadata": {},
   "source": [
    "# Exercice | Semi-Supervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a33c3fc",
   "metadata": {},
   "source": [
    "# 🚀️Goals\n",
    "- Learn the implementation of semi-supervised learning\n",
    "- Experiment with different methods\n",
    "- Get an impression on their pros and cons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b6ba417",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.semi_supervised import LabelSpreading \n",
    "from sklearn.semi_supervised import SelfTrainingClassifier \n",
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b764271",
   "metadata": {},
   "source": [
    "In this exercice we will use the digits dataset from scikit learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30e3adf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the digits dataset\n",
    "data = load_digits()\n",
    "\n",
    "# Split the dataset into training and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2, random_state=42)\n",
    "\n",
    "print('Shape dataset training : ' ,X_train.shape[0],' and testing :', X_test.shape[0])\n",
    "print('Feature shape : ',X_train.shape[1])\n",
    "print('Classes in the dataset :',np.unique(y_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dab6564",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply standard scaling to the training and test sets\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "X_test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c00e5cec",
   "metadata": {},
   "source": [
    "# Semi-supervised Learning (simple approach)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59739e8c",
   "metadata": {},
   "source": [
    "In this exercise, you will implement a vanilla semi-supervised learning. You will train a model on a subset of the labeled data, make predictions on the unlabeled data, and then train a second model on both the labeled and pseudo-labeled data. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d009846",
   "metadata": {},
   "source": [
    "## 1) Train only a small sample of labeled data and perform evaluation on the testing set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13b2313e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the LabelSpreading model\n",
    "lp_model = LabelSpreading()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6cd26e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Train a lp_model model on the first 100 labeled samples:\n",
    "\n",
    "'''\n",
    "YOUR CODE HERE\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ea62035",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Test the model's accuracy on the testing set\n",
    "test_acc = lp_model.score(X_test_scaled, y_test)\n",
    "print(\"Testing set accuracy:\", test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bb365e3",
   "metadata": {},
   "source": [
    "## 2) Use the model to make prediction on the unlabled samples (not first 100 samples of X_train_scaled) and then retrain a new model with the new 'labels' and evaluate the performances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d14e146",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions on the unlabeled samples and combine them with the labeled samples:\n",
    "'''\n",
    "YOUR CODE HERE\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39d5b3db",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Retrain the lp_model on the combined set of labeled and pseudo-labeled samples:\n",
    "'''\n",
    "YOUR CODE HERE\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d78c5df7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Evaluate the model's accuracy on the testing set again:\n",
    "test_acc = lp_model.score(X_test_scaled, y_test)\n",
    "print(\"Testing set accuracy:\", test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5737b85f",
   "metadata": {},
   "source": [
    "# Semi-supervised Learning with Label Propagation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91044696",
   "metadata": {},
   "source": [
    "Label Propagation is a semi-supervised learning algorithm that can be used to predict labels for data points in a dataset where only a small subset of the data has labeled labels. The algorithm assumes that data points that are close together in the feature space are likely to belong to the same class. It works by propagating the labels of labeled data points to nearby unlabeled data points.\n",
    "\n",
    "The algorithm starts by assigning the labeled data points their true labels, and the unlabeled data points arbitrary labels. Then, it iteratively propagates the labels of labeled data points to their neighbors in the feature space, using a Gaussian kernel to weight the influence of each labeled point on its neighbors. After each iteration, the algorithm updates the labels of the unlabeled points based on the labels of their neighbors, and continues until convergence."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "064024ea",
   "metadata": {},
   "source": [
    "Now you will implement the Label Propagation algorithm. We'll need to pass now the unlabled data points directly to the method, however the labels will be set to -1 for those one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31d73868",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the LabelSpreading model\n",
    "lp_model = LabelSpreading()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1bd4ab6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the labeled and unlabeled data to fit the model\n",
    "y_labeled = y_train.copy()\n",
    "y_labeled[100:] = -1  # Set labels of the unlabeled data to -1\n",
    "\n",
    "'''\n",
    "YOUR CODE HERE to fit the model \n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df92b498",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test the model's accuracy on the testing set\n",
    "test_acc = lp_model.score(X_test_scaled, y_test)\n",
    "print(\"Label Spreading Testing set accuracy:\", test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c039b407",
   "metadata": {},
   "source": [
    "Compare the results with previous methods (supervised learning on only the first 100 samples and the vanilla semi-supervised method. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "420c5e91",
   "metadata": {},
   "source": [
    "## We can also visualize the performance of the model by plotting a graph of the accuracy on the test set versus the number of labeled data points used to train the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ae456fe",
   "metadata": {},
   "source": [
    "Here, we start by training the model on just 10% labeled data points and then gradually increase the number of labeled data points by 10% at a time. For each number of labeled data points, we retrain the semi-supervised and supervised models and evaluate their accuracy on the test set. We then plot a graph of the accuracy on the test set versus the number of labeled data points used to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c2701ec1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vary the number of labeled samples from 10% to 90% of the total number of samples\n",
    "percentages = list(range(10, 100, 10))\n",
    "test_accs_semi = []\n",
    "test_accs = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a887a90c",
   "metadata": {},
   "outputs": [],
   "source": [
    "for percentage in percentages:\n",
    "    # Split the training set into labeled and unlabeled data\n",
    "    X_labeled, X_unlabeled, y_labeled, y_unlabeled = train_test_split(X_train, y_train, train_size=percentage/100, stratify=y_train, random_state=42)\n",
    "    \n",
    "    # Apply standard scaling to the labeled and unlabeled data\n",
    "    X_labeled_scaled = scaler.transform(X_labeled)\n",
    "    X_unlabeled_scaled = scaler.transform(X_unlabeled)\n",
    "\n",
    "    # Use the labeled and unlabeled data to fit the model\n",
    "    \n",
    "    '''\n",
    "    YOUR CODE HERE\n",
    "    '''\n",
    "    \n",
    "    # Test the model's accuracy on the testing set\n",
    "    test_acc = lp_model.score(X_test_scaled, y_test)\n",
    "    test_accs_semi.append(test_acc)\n",
    "    \n",
    "    #Compare semi-supervised learning to supervised learning\n",
    "    lp_model.fit(X_labeled_scaled, y_labeled)\n",
    "\n",
    "    # Test the model's accuracy on the testing set\n",
    "    test_acc = lp_model.score(X_test_scaled, y_test)\n",
    "    test_accs.append(test_acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab17209a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the accuracy on the test set for each percentage of labeled samples\n",
    "plt.plot(percentages, test_accs,label='Supervised learning')\n",
    "plt.plot(percentages, test_accs_semi,label='Label Propagation')\n",
    "plt.title(\"Label Spreading Accuracy on Test Set vs Percentage of Labeled Samples\")\n",
    "plt.xlabel(\"Percentage of Labeled Samples\")\n",
    "plt.ylabel(\"Accuracy on Test Set\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7af9681",
   "metadata": {},
   "source": [
    "# Semi-supervised Learning with Self Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c8602ef",
   "metadata": {},
   "source": [
    "Self Training is another semi-supervised learning algorithm that utilizes a classifier to label the unlabeled data. The basic idea behind this algorithm is that it trains a classifier on the labeled data and then uses this classifier to predict labels for the unlabeled data. These predicted labels are then used to expand the labeled dataset, and the process is repeated iteratively.\n",
    "\n",
    "The steps involved in using Self Training are:\n",
    "\n",
    "1) Initialize the classifier and fit it on the labeled data.\n",
    "2) Predict labels for the unlabeled data using the trained classifier.\n",
    "3) Add the predicted labels to the labeled data and retrain the classifier.\n",
    "4) Repeat steps 2 and 3 until convergence or until a maximum number of iterations is reached."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdbfc282",
   "metadata": {},
   "source": [
    "Fit an SVM classifier on the first 100 labeled samples using the vanilla supervised approach and calculate the test accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58b2370d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up an SVM classifier \n",
    "lp_model = SVC(probability=True)\n",
    "\n",
    "'''\n",
    "YOUR CODE HERE\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57fae2bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test the model's accuracy on the testing set\n",
    "test_acc = lp_model.score(X_test_scaled, y_test)\n",
    "print(\"Testing set accuracy:\", test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82e8aff0",
   "metadata": {},
   "source": [
    "Fit a SelfTrainingClassifier on the entire labeled dataset with the same SVM classifier and calculate the test accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6556b3aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the labeled data to fit the model\n",
    "y_labeled = y_train.copy()\n",
    "y_labeled[100:] = -1  # Set labels of the unlabeled data to -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edf5af56",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "YOUR CODE HERE\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b51c113",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test the model's accuracy on the testing set\n",
    "test_acc = self_training_model.score(X_test_scaled, y_test)\n",
    "print(\"Self Training Testing set accuracy:\", test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "061bc076",
   "metadata": {},
   "source": [
    "## We can also visualize the performance of the model by plotting a graph of the accuracy on the test set versus the number of labeled data points used to train the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86ae4edb",
   "metadata": {},
   "source": [
    "Similar to above, but with SelfTrainingClassifier "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7de6cf21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vary the number of labeled samples from 10% to 90% of the total number of samples\n",
    "percentages = list(range(10, 100, 10))\n",
    "test_accs_semi = []\n",
    "test_accs = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d662bd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "for percentage in percentages:\n",
    "    # Split the training set into labeled and unlabeled data\n",
    "    X_labeled, X_unlabeled, y_labeled, y_unlabeled = train_test_split(X_train, y_train, train_size=percentage/100, stratify=y_train, random_state=42)\n",
    "    \n",
    "    # Apply standard scaling to the labeled and unlabeled data\n",
    "    X_labeled_scaled = scaler.transform(X_labeled)\n",
    "    X_unlabeled_scaled = scaler.transform(X_unlabeled)\n",
    "    \n",
    "    '''\n",
    "    YOUR CODE HERE\n",
    "    '''\n",
    "    \n",
    "    # Test the model's accuracy on the testing set\n",
    "    test_acc = self_training_model.score(X_test_scaled, y_test)\n",
    "    test_accs_semi.append(test_acc)\n",
    "    \n",
    "    lp_model.fit(X_labeled_scaled, y_labeled)\n",
    "\n",
    "    # Test the model's accuracy on the testing set\n",
    "    test_acc = lp_model.score(X_test_scaled, y_test)\n",
    "    test_accs.append(test_acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42a70d2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the accuracy on the test set for each percentage of labeled samples\n",
    "plt.plot(percentages, test_accs,label='Supervised learning')\n",
    "plt.plot(percentages, test_accs_semi,label='Self Training')\n",
    "plt.title(\"Self Training Accuracy on Test Set vs Percentage of Labeled Samples\")\n",
    "plt.xlabel(\"Percentage of Labeled Samples\")\n",
    "plt.ylabel(\"Accuracy on Test Set\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}